India's unorganized sector, which is made up of millions of small businesses that are privately owned, make up about 93% of the country's total workforce.
As many as 32% of the respondents said increasing unemployment was the key reason why they would not elect the BJP again.
India's Labour Ministry did not immediately respond to CNBC's queries pertaining to the country's unemployment situation.
Rajan, who was speaking about how to make India an advanced economy at the George Washington University, said: "Unemployment numbers are high, disguised unemployment is even higher.
A slowdown in hiring in India's huge information technology sector is also to blame for the lack of well-paying, white-collar jobs.
Persons:
Rahul Gandhi, Narendra Modi, Modi, Gandhi, —, Manmohan Singh, Arun Kumar, Lokniti, joblessness, Kumar, Raghuram Rajan, Rajan
Organizations:
DELHI, International Labour Organisation, Institute of Human, Goods, Services Tax, Jawahar Lal Nehru, CNBC, ILO, Bharatiya Janata Party, Indian National Congress, BJP, Modi, India's Labour Ministry, Former Reserve Bank of India, George Washington University, Labor
Locations:
India, Bihar, New Delhi